3.4 Useful Techniques when Writing Tests

Testing simple functions that have no side effects and no dependencies on their environment is easy. Such tests often look like this:

(ert-deftest ert-test-mismatch ()
  (should (eql (ert--mismatch "" "") nil))
  (should (eql (ert--mismatch "" "a") 0))
  (should (eql (ert--mismatch "a" "a") nil))
  (should (eql (ert--mismatch "ab" "a") 1))
  (should (eql (ert--mismatch "Aa" "aA") 0))
  (should (eql (ert--mismatch '(a b c) '(a b d)) 2)))

This test calls the function ert--mismatch several times with various combinations of arguments and compares the return value to the expected return value. (Some programmers prefer (should (eql EXPECTED ACTUAL)) over the (should (eql ACTUAL EXPECTED)) shown here. ERT works either way.)

Here’s a more complicated test:

(ert-deftest ert-test-record-backtrace ()
  (let ((test (make-ert-test :body (lambda () (ert-fail "foo")))))
    (let ((result (ert-run-test test)))
      (should (ert-test-failed-p result))
      (with-temp-buffer
        (ert--print-backtrace (ert-test-failed-backtrace result))
        (goto-char (point-min))
        (end-of-line)
        (let ((first-line (buffer-substring-no-properties
                           (point-min) (point))))
          (should (equal first-line
                         "  signal(ert-test-failed (\"foo\"))")))))))

This test creates a test object using make-ert-test whose body will immediately signal failure. It then runs that test and asserts that it fails. Then, it creates a temporary buffer and invokes ert--print-backtrace to print the backtrace of the failed test to the current buffer. Finally, it extracts the first line from the buffer and asserts that it matches what we expect. It uses buffer-substring-no-properties and equal to ignore text properties; for a test that takes properties into account, buffer-substring and ert-equal-including-properties could be used instead.

The reason why this test only checks the first line of the backtrace is that the remainder of the backtrace is dependent on ERT’s internals as well as whether the code is running interpreted or compiled. By looking only at the first line, the test checks a useful property—that the backtrace correctly captures the call to signal that results from the call to ert-fail—without being brittle.

This example also shows that writing tests is much easier if the code under test was structured with testing in mind.

For example, if ert-run-test accepted only symbols that name tests rather than test objects, the test would need a name for the failing test, which would have to be a temporary symbol generated with make-symbol, to avoid side effects on Emacs’s state. Choosing the right interface for ert-run-tests allows the test to be simpler.

Similarly, if ert--print-backtrace printed the backtrace to a buffer with a fixed name rather than the current buffer, it would be much harder for the test to undo the side effect. Of course, some code somewhere needs to pick the buffer name. But that logic is independent of the logic that prints backtraces, and keeping them in separate functions allows us to test them independently.

A lot of code that you will encounter in Emacs was not written with testing in mind. Sometimes, the easiest way to write tests for such code is to restructure the code slightly to provide better interfaces for testing. Usually, this makes the interfaces easier to use as well.